Note: interpreting iterative methods convergence 
with diffusion point of view 
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Abstract — In this paper, we explain the convergence speed of 
different iteration schemes with the fluid diffusion view when 
solving a hnear fixed point problem. This interpretation allows 
one to better understand why power iteration or Jacobi iteration 
may converge faster or slower than Gauss-Seidel iteration. 
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L Introduction 

Based on the previous research results on the diffusion 
approach IT] to solve fixed point problem in linear algebra, 
we propose here a new analysis of the convergence speed of 
different iteration methods. 

In Section [III we define the iteration methods that are 
considered. Section |III] shows how to define the associated 
equivalent diffusion iteration. Section |IV] shows few examples 
to illustrate the application. 

II. Algorithms description 

A. Notations 

We will use the following notations: 

. P G IR^"^^ a real matrix; 

. I e ]R^^^ the identity matrix; 

• Ji the matrix with all entries equal to zero except for the 
i-th diagonal term: {3i)ii ~ 1; 

. n = {!,.., iV}; 

. cr : R^ ^ R defined by a{X) = ^^^ x,; 

• I = {ii, 12, .., i„, ...} a sequence of coordinate: ik S fi; 

• a fair sequence is a sequence where all elements of fi 
appears infinitely often; 

. e = (l/7V,..,l/Af)^. 

B. Problem to solve 

We will consider two types of linear fixed point problems: 

X = VX 

X e R" and: 

X = I'X + B 



C. Linear equation: X = PX 

1) Power iteration (PI): The power iteration PI{P,Xo) is 
defined by: 

Xn ~ 'PXn-\ (1) 

Starting from X^. 

2) Gauss-Seidel iteration (GSl): Given a sequence of nodes 
for the updatel = {ii, i2, .., in, ■■■}, the Gauss-Seidel iteration 
GSl(P,Xo,I) is defined by: 

(X),„ = (PX),„ (2) 

starting from Xq, which simply means that the n-th update on 
X is on coordinate i„ based on the last vector X (each update 
modifying only one coordinate of X). We could equivalently 
write it as: 

Xn — Xn-1 + Ji,i(P ~ I)-''^n-l- 

D. Affine equation: X = PX + B 

1) Jacobi iteration (Jac): The Jacobi iteration J{P,B) is 
defined by: 



Xn — PX„ 



B 



(3) 



B gr 



N 



starting from Xq = (0, .., 0)^. 

2) Gauss-Seidel iteration (GSa): Given a sequence of 
nodes for the update I = {«i,«2, ••,«n, ■••}, the Gauss-Seidel 
iteration GSa(P,B,I) is defined by: 

(X),„ = (PX„-i + B),„ (4) 

starting from Xq — (0, .., 0)-'". We could equivalently write it 
as: 

Xn = (I - J»JX„_i + J,„ (PX„_i + B). 

Note that in the above notations, the n of Xn may mean 
the n-th application of an operator on a vector (Jacobi style, 
vector level update or VLU) or the n-th update of a coordinate 
of X (Gauss-Seidel style, coordinate level update or CLU). In 
terms of diffusion view, the VLU can be always interpreted 
as a partial diffusion of CLU approach cf. JS). 

Remark 1: One could also consider a more general iteration 
scheme such as BiCGSTAB or GMRES, but they require a 
fluid injection method, which may be more complex. This is 
let for a future research. 



Type 


VLU 


CLU 


VLU 


CLU 


Scheme 


PI 


GSl 


Jac 


GSa 


Defined by 


P,X„ 


P,Xo,2 


P,B 


P,B,I 


Diffusion 


P,PXo-Xo 


P,PXo-Xo,X 


P,B 


P,B,I. 



TABLE I 
Iteration scheme equivalence. 



III. Diffusion equations 

The diffusion equations are defined by two state vectors 
Fn and _ff„ associated to the affine equation X = PX + B 
(DI{F,B,I)): 

Fn = ^„-i + (P-I)J.„^„_i, (5) 

The VLU adaptation of the above equation would be: 

Fn = PFn-l, (6) 

Hn = Hn-l+ Fn-l- 

which is equivalent to 

Hn = PHn-i + B, 
which is a Jacobi iteration. 

A. Linear equation: X = PX 

1) Power iteration (PI): We define: Fq = PXq-Xq, Hq = 
and the iterative diffusion equation with VLU: 

Fn = PFn^l, 
Hn = Hn^l + Fn-l- 

Then, we have the equalities: 

Fn = -'fri+l — Xn, 
Hn = Xn — Xq. 

2) Gauss-Seidel iteration (GSl): We set Fq = PXq — Xq, 
Hq = and iJ„,F„ defined by Equations (|5]l, then we have 

Xn = Hn + Xq. 

B. Affine equation: X ^ PX + B 

1) Jacobi iteration (Jac): The Jacobi iteration is equivalent 
to the VLU of diffusion equations |6] 

2) Gauss-Seidel iteration (GSa): The Gauss-Seidel itera- 
tion is equivalent to the (CLU of) diffusion equations |5] 

C. Summary of results on the equivalent iterations 

IV. Example of convergence speed comparison 
For comparison analysis, we consider a PageRank equation 
il: 

X = PX = {dQ + {l- d)/NJ)X = dqX + (1 - d)e 

assuming <j{X) = 1. Then we have: 

. PI{P, Xo) = DI{dQ, dQXo + (1 - d)e - Xq) + Xq- 



. Jac{dQ, (1 - d)e) = DI{dQ, (1 - rf)e); 

. GSa{dQ, (1 - d)e,I) = DI{dQ, (1 - d)e,I); 

. GSl{P, Xq = e, J) = DlidQ, d{Qe - e),I). 

Note that GSl{P,Xo,I) has no reason to converge in 
general. II will converge if I is a negative or positive fair 
sequence cf. JS). Here, the decomposition of P guarantees 
that it always converges for any fair sequence I. Note also 
that Jac{dQ, (1 - d)e) and GSa{dQ, (1 - d)e,I) define a 
non-decreasing vector X„ (positive fluid diffusion). 

Below, we compare PI, Jac and GS in simple scenarios. 

A. Case 1 

Note that the convergence is below measured in residual 
fluid using the Li norm of Fn of the equivalent diffusion 
equation. For an easy comparison, the x-axis shows the number 
of iterations: for VLU approaches, it is exactly the index n of 
Fn, for CLU approaches, the Li norm of F^^n is shown. We 
take d = 0.85 and 



Q 



/o 











0.5\ 


1 











0.5 





1 

















1 








Vo 








1 


0/ 




Nb iterations 

Fig. L Convergence. 

Figure [T] Jacobi iteration converges exactly as d". PI does 
roughly the same. GSa is much faster because we follow the 
graph path and send cumulated fluid: it is roughly 4-5 times 
faster as expected. GSl has the convergence slope of GSa but 
with a larger jump at the first iteration. 



B. Case 2 
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Figure 12] Jacobi iteration converges still as d". PI and GSa 
have similar convergence slope. GSl still shows the first jump. 



'C 














Pl — 1 — ■ 

Jacobi 
GS-A -+ - 
I 0.85^x 














^ X 


\ 


^^A 


















^ 






























yi 


\ 














■* 


\ 


"^ 














* 




V-^ 












>«, 
















^ 


>_ 


>. 


\ 


V- 


-^ 



Nb iterations 



0.1 


+ 


* 


* 


\ 
















PI 

Jacobi 

GS-A 

I 0.85^x 


■^-■ 




» 


* 


J\ 


\ 


* 




" * 










0.001 










^^ 


\ 






* 




^ ^ ■* 


* ■■* ■ 










































X 


























^ \ ^ 










1fi-07 


\ 



10 15 20 25 

Nb iterations 



Fig. 2. Convergence. 



Fig. 4. Convergence. 



C. Case 3 
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Fig. 3. Convergence. 

Figure |3] Jacobi iteration converges always as d". PI starts 
to converge faster than GSa when we add more links. GSl has 
the first jump. 

To explain, this convergence speed difference between the 
four methods, we consider the case 4 below. 



D. Case 4 
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Figure |4l with GSl, after one iteration (5 updates), we have 



positive fluid (node 5) at 5-th update. Then, from 6-th update, 
we only move positive fluid. And only a fraction of 1% at 
node 5 are moved: this explains the convergence at d" after 
iteration 1 for GSa and GSl. For cases 1-3, the whole fluid 
(no self loop) from each node is moved to children nodes and 
this explains the gain factor (merging fluid before moving). 

Now the good performance of PI can be explained by 
the fact that doing partial diffusion we create more fluid 
cancellation and make the convergence faster than d". To 
confirm this explanation, we plot in Figure |5] the amount 
of fluid that has been canceled at each iteration. We see 
that this phenomenon is driving the convergence speed (the 
difference to the residual fluid is due to the contracting factor 
that eliminates at each diffusion a fraction 1 — d (15% here). 
In this case, the fluid disappeared after 5-th update 84% due 
to fluid cancellation and 16% due to the contracting factor. 
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Fig. 5. Convergence: cancelled fluid. 



V. Conclusion 

In this paper, we described the equivalent equations of the 
diffusion iteration associated to power iteration, Jacobi and 
Gauss-Seidel iteration and showed how they can explain the 



a big jump due to the cumulated negative fluid that meets the convergence speed of each method. 
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